Binaural horizontal perspective display

ABSTRACT

The present invention display system discloses a three dimension display system comprising a three dimensional horizontal perspective display and a 3-D audio system such as binaural simulation to lend realism to the three dimensional display. The three dimensional display system can futher comprise a second display, together with a curvilinear blending display section to merge the various images. The multi-plane display surface can accommodate the viewer by adjusting the various images and 3-D sound according to the viewer&#39;s eyepoint and earpoint locations.

This application claims priority from U.S. provisional applications Ser.No. 60/576,187 filed Jun. 1, 2004, entitled “Multi plane horizontalperspective display”; Ser. No. 60/576,189 filed Jun. 1, 2004, entitled“Multi plane horizontal perspective hand on simulator”; Ser. No.60/576,182 filed Jun. 1, 2004, entitled “Binaural horizontal perspectivedisplay”; and Ser. No. 60/576,181 filed Jun. 1, 2004, entitled “Binauralhorizontal perspective hand on simulator” which are incorporated hereinby reference.

This application is related to co-pending application Ser. No.11/098,681 filed Apr. 4, 2005, entitled “Horizontal projection display”;Ser. No. 11/098,685 filed Apr. 4, 2005, entitled “Horizontal projectiondisplay”, Ser. No. 11/098,667 filed Apr. 4, 2005, entitled “Horizontalprojection hands-on simulator”; Ser. No. 11/098,682 filed Apr. 4, 2005,entitled “Horizontal projection hands-on simulator”; “Multi planehorizontal perspective display” filed May 27, 2005; “Multi planehorizontal perspective hand on simulator” filed May 27, 2005; “Binauralhorizontal perspective display” filed May 27, 2005; and “Binauralhorizontal perspective hand on simulator” filed May 27, 2005.

FIELD OF INVENTION

This invention relates to a three-dimensional display system, and inparticular, to a multiple view display system.

BACKGROUND OF THE INVENTION

Ever since humans began to communicate through pictures, they faced adilemma of how to accurately represent the three-dimensional world theylived in. Sculpture was used to successfully depict three-dimensionalobjects, but was not adequate to communicate spatial relationshipsbetween objects and within environments. To do this, early humansattempted to “flatten” what they saw around them onto two-dimensional,vertical planes (e.g. paintings, drawings, tapestries, etc.). Sceneswhere a person stood upright, surrounded by trees, were renderedrelatively successfully on a vertical plane. But how could theyrepresent a landscape, where the ground extended out horizontally fromwhere the artist was standing, as far as the eye could see?

The answer is three dimensional illusions. The two dimensional picturesmust provide a numbers of cues of the third dimension to the brain tocreate the illusion of three dimensional images. This effect of thirddimension cues can be realistically achievable due to the fact that thebrain is quite accustomed to it. The three dimensional real world isalways and already converted into two dimensional (e.g. height andwidth) projected image at the retina, a concave surface at the back ofthe eye. And from this two dimensional image, the brain, throughexperience and perception, generates the depth information to form thethree dimension visual image from two types of depth cues: monocular(one eye perception) and binocular (two eye perception). In general,binocular depth cues are innate and biological while monocular depthcues are learned and environmental.

The major binocular depth cues are convergence and retinal disparity.The brain measures the amount of convergence of the eyes to provide arough estimate of the distance since the angle between the line of sightof each eye is larger when an object is closer. The disparity of theretinal images due to the separation of the two eyes is used to createthe perception of depth. The effect is called stereoscopy where each eyereceives a slightly different view of a scene, and the brain fuses themtogether using these differences to determine the ratio of distancesbetween nearby objects.

Binocular cues are very powerful perception of depth. However, there arealso depth cues with only one eye, called monocular depth cues, tocreate an impression of depth on a flat image. The major monocular cuesare: overlapping, relative size, linear perspective and light andshadow. When an object is viewed partially covered, this pattern ofblocking is used as a cue to determine that the object is farther away.When two objects known to be the same size and one appears smaller thanthe other, this pattern of relative size is used as a cue to assume thatthe smaller object is farther away. The cue of relative size alsoprovides the basis for the cue of linear perspective where the fartheraway the lines are from the observer, the closer together they willappear since parallel lines in a perspective image appear to convergetowards a single point. The light falling on an object from a certainangle could provide the cue for the form and depth of an object. Thedistribution of light and shadow on a objects is a powerful monocularcue for depth provided by the biologically correct assumption that lightcomes from above.

Perspective drawing, together with relative size, is most often used toachieve the illusion of three dimension depth and spatial relationshipson a flat (two dimension) surface, such as paper or canvas. Throughperspective, three dimension objects are depicted on a two dimensionplane, but “trick” the eye into appearing to be in three dimensionspace. The first theoretical treatise for constructing perspective,Depictura, was published in the early 1400's by the architect, LeoneBattista Alberti. Since the introduction of his book, the details behind“general” perspective have been very well documented. However, the factthat there are a number of other types of perspectives is not wellknown. Some examples are military, cavalier, isometric, and dimetric, asshown at the top of FIG. 1.

Of special interest is the most common type of perspective, calledcentral perspective, shown at the bottom left of FIG. 1. Centralperspective, also called one-point perspective, is the simplest kind of“genuine” perspective construction, and is often taught in art anddrafting classes for beginners. FIG. 2 further illustrates centralperspective. Using central perspective, the chess board and chess pieceslook like three dimension objects, even though they are drawn on a twodimensional flat piece of paper. Central perspective has a centralvanishing point, and rectangular objects are placed so their front sidesare parallel to the picture plane. The depth of the objects isperpendicular to the picture plane. All parallel receding edges runtowards a central vanishing point. The viewer looks towards thisvanishing point with a straight view. When an architect or artistcreates a drawing using central perspective, they must use a single-eyeview. That is, the artist creating the drawing captures the image bylooking through only one eye, which is perpendicular to the drawingsurface.

The vast majority of images, including central perspective images, aredisplayed, viewed and captured in a plane perpendicular to the line ofvision. Viewing the images at angle different from 90° would result inimage distortion, meaning a square would be seen as a rectangle when theviewing surface is not perpendicular to the line of vision. However,there is a little known class of images that we called it “horizontalperspective” where the image appears distorted when viewing head on, butdisplaying a three dimensional illusion when viewing from the correctviewing position. In horizontal perspective, the angle between theviewing surface and the line of vision is preferrably 45° but can bealmost any angle, and the viewing surface is perferrably horizontal(wherein the name “horizontal perspective”), but it can be any surface,as long as the line of vision forming a not-perpendicular angle to it.

Horizontal perspective images offer realistic three dimensionalillusion, but are little known primarily due to the narrow viewinglocation (the viewer's eyepoint has to be coincide precisely with theimage projection eyepoint), and the complexity involving in projectingthe two dimensional image or the three dimension model into thehorizontal perspective image.

The generation of horizontal perspective images require considerablymore expertise to create than conventional perpendicular images. Theconventional perpendicular images can be produced directly from theviewer or camera point. One need simply open one's eyes or point thecamera in any direction to obtain the images. Further, with muchexperience in viewing three dimensional depth cues from perpendicularimages, viewers can tolerate significant amount of distortion generatedby the deviations from the camera point. In contrast, the creation of ahorizontal perspective image does require much manipulation.Conventional camera, by projecting the image into the planeperpendicular to the line of sight, would not produce a horizontalperspective image. Making a horizontal drawing requires much effort andvery time consuming. Further, since human has limited experience withhorizontal perspective images, the viewer's eye must be positionedprecisely where the projection eyepoint point is to avoid imagedistortion. And therefore horizontal perspective, with its difficulties,has received little attention.

For realistic three dimensional display, binaural or three dimensionalaudio simulation is also needed.

SUMMARY OF THE INVENTION

The present invention recognizes that the personal computer is perfectlysuitable for horizontal perspective display. It is personal, thus it isdesigned for the operation of one person, and the computer, with itspowerful microprocessor, is well capable of rendering various horizontalperspective images to the viewer.

Thus the present invention display system discloses a three dimensiondisplay system comprising at least a display surface displaying a threedimensional horizontal perspective images. The other display surfacescan display two dimensional images, or preferably three dimensionalcentral perpective images. Further, the display surfaces can have acurvilinear blending display section to merge the various images. Themulti-plane display system can comprise various camera eyepoints, onefor the horizontal perspective images, one for the central perspectiveimages, and optionally one for the curvilinear blending display surface.The multi-plane display surface can further adjust the various images toaccommodate the position of the viewer. By changing the displayed imagesto keep the camera eyepoints of the horizontal perspective and centralperspective images in the same position as the viewer's eye point, theviewer's eye is always positioned at the proper viewing position toperceive the three dimensional illusion, thus minimizing viewer'sdiscomfort and distortion. The display can accept manual input such as acomputer mouse, trackball, joystick, tablet, etc. to re-position thehorizontal perspective images. The display can also automaticallyre-position the images based on an input device automatically providingthe viewer's viewpoint location.

Further, the display is also included three dimensional audio such asbinaural simulation to lend realism to the three dimensional display.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows the various perspective drawings.

FIG. 2 shows a typical central perspective drawing.

FIG. 3 shows the comparison of central perspective (Image A) andhorizontal perspective (Image B).

FIG. 4 shows the central perspective drawing of three stacking blocks.

FIG. 5 shows the horizontal perspective drawing of three stackingblocks.

FIG. 6 shows the method of drawing a horizontal perspective drawing.

FIG. 7 shows a horizontal perspective display and an viewer inputdevice.

FIG. 8 shows a horizontal perspective display, a computational deviceand an viewer input device.

FIG. 9 shows mapping of the 3-d object onto the horizontal plane.

FIG. 10 shows the projection of 3-d object by horizontal perspective.

FIG. 11 shows the simulation time of the horizontal perspective.

FIG. 12 shows an embodiment of the present invention multi-planedisplay.

FIG. 13 shows the horizontal perspective and central perspectiveprojection on the present invention multi-plane display.

DETAILED DESCRIPTION OF THE INVENTION

The present invention discloses a multi-plane display system comprisingat least two display surfaces, one of which capable of projecting threedimensional illusion based on horizontal perspective projection.

In general, the present invention multi-plane display system can be usedto display three dimensional images and has obvious utility to manyindustrial applications such as manufacturing design reviews, ergonomicsimulation, safety and training, video games, cinematography, scientific3D viewing, and medical and other data displays.

Horizontal perspective is a little-known perspective, of which we foundonly two books that describe its mechanics: Stereoscopic Drawing (©1990)and How to Make Anaglyphs (©1979, out of print). Although these booksdescribe this obscure perspective, they do not agree on its name. Thefirst book refers to it as a “free-standing anaglyph,” and the second, a“phantogram.” Another publication called it “projective anaglyph” (U.S.Pat. No. 5,795,154 by G. M. Woods, Aug. 18, 1998). Since there is noagreed-upon name, we have taken the liberty of calling it “horizontalperspective.” Normally, as in central perspective, the plane of vision,at right angle to the line of sight, is also the projected plane of thepicture, and depth cues are used to give the illusion of depth to thisflat image. In horizontal perspective, the plane of vision remains thesame, but the projected image is not on this plane. It is on a planeangled to the plane of vision. Typically, the image would be on theground level surface. This means the image will be physically in thethird dimension relative to the plane of vision. Thus horizontalperspective can be called horizontal projection.

In horizontal perspective, the object is to separate the image from thepaper, and fuse the image to the three dimension object that projectsthe horizontal perspective image. Thus the horizontal perspective imagemust be distorted so that the visual image fuses to form the freestanding three dimensional figure. It is also essential the image isviewed from the correct eye points, otherwise the three dimensionalillusion is lost. In contrast to central perspective images which haveheight and width, and project an illusion of depth, and therefore theobjects are usually abruptly projected and the images appear to be inlayers, the horizontal perspective images have actual depth and width,and illusion gives them height, and therefore there is usually agraduated shifting so the images appear to be continuous.

FIG. 3 compares key characteristics that differentiate centralperspective and horizontal perspective. Image A shows key pertinentcharacteristics of central perspective, and Image B shows key pertinentcharacteristics of horizontal perspective.

In other words, in Image A, the real-life three dimension object (threeblocks stacked slightly above each other) was drawn by the artistclosing one eye, and viewing along a line of sight perpendicular to thevertical drawing plane. The resulting image, when viewed vertically,straight on, and through one eye, looks the same as the original image.

In Image B, the real-life three dimension object was drawn by the artistclosing one eye, and viewing along a line of sight 45° to the horizontaldrawing plane. The resulting image, when viewed horizontally, at 45° andthrough one eye, looks the same as the original image.

One major difference between central perspective showing in Image A andhorizontal perspective showing in Image B is the location of the displayplane with respect to the projected three dimensional image. Inhorizontal perspective of Image B, the display plane can be adjusted upand down, and therefore the projected image can be displayed in the openair above the display plane, i.e. a physical hand can touch (or morelikely pass through) the illusion, or it can be displayed under thedisplay plane, i.e. one cannot touch the illusion because the displayplane physically blocks the hand. This is the nature of horizontalperspective, and as long as the camera eyepoint and the viewer eyepointis at the same place, the illusion is present. In contrast, in centralperspective of Image A, the three dimensional illusion is likely to beonly inside the display plane, meaning one cannot touch it. To bring thethree dimensional illusion outside of the display plane to allow viewerto touch it, the central perspective would need elaborate display schemesuch as surround image projection and large volume.

FIGS. 4 and 5 illustrate the visual difference between using central andhorizontal perspective. To experience this visual difference, first lookat FIG. 4, drawn with central perspective, through one open eye. Holdthe piece of paper vertically in front of you, as you would atraditional drawing, perpendicular to your eye. You can see that centralperspective provides a good representation of three dimension objects ona two dimension surface.

Now look at FIG. 5, drawn using horizontal perspective, by sifting atyour desk and placing the paper lying flat (horizontally) on the desk infront of you. Again, view the image through only one eye. This puts yourone open eye, called the eye point at approximately a 45° angle to thepaper, which is the angle that the artist used to make the drawing. Toget your open eye and its line-of-sight to coincide with the artist's,move your eye downward and forward closer to the drawing, about sixinches out and down and at a 45° angle. This will result in the idealviewing experience where the top and middle blocks will appear above thepaper in open space.

Again, the reason your one open eye needs to be at this precise locationis because both central and horizontal perspective not only define theangle of the line of sight from the eye point; they also define thedistance from the eye point to the drawing. This means that FIGS. 4 and5 are drawn with an ideal location and direction for your open eyerelative to the drawing surfaces. However, unlike central perspectivewhere deviations from position and direction of the eye point createlittle distortion, when viewing a horizontal perspective drawing, theuse of only one eye and the position and direction of that eye relativeto the viewing surface are essential to seeing the open space threedimension horizontal perspective illusion.

FIG. 6 is an architectural-style illustration that demonstrates a methodfor making simple geometric drawings on paper or canvas utilizinghorizontal perspective. FIG. 6 is a side view of the same three blocksused in FIG. 5. It illustrates the actual mechanics of horizontalperspective. Each point that makes up the object is drawn by projectingthe point onto the horizontal drawing plane. To illustrate this, FIG. 6shows a few of the coordinates of the blocks being drawn on thehorizontal drawing plane through projection lines. These projectionlines start at the eye point (not shown in FIG. 6 due to scale),intersect a point on the object, then continue in a straight line towhere they intersect the horizontal drawing plane, which is where theyare physically drawn as a single dot on the paper When an architectrepeats this process for each and every point on the blocks, as seenfrom the drawing surface to the eye point along the line-of-sight thehorizontal perspective drawing is complete, and looks like FIG. 5.

Notice that in FIG. 6, one of the three blocks appears below thehorizontal drawing plane. With horizontal perspective, points locatedbelow the drawing surface are also drawn onto the horizontal drawingplane, as seen from the eye point along the line-of-site. Therefore whenthe final drawing is viewed, objects not only appear above thehorizontal drawing plane, but may also appear below it as well-givingthe appearance that they are receding into the paper. If you look againat FIG. 5, you will notice that the bottom box appears to be below, orgo into, the paper, while the other two boxes appear above the paper inopen space.

The generation of horizontal perspective images require considerablymore expertise to create than central perspective images. Even thoughboth methods seek to provide the viewer the three dimension illusionthat resulted from the two dimensional image, central perspective imagesproduce directly the three dimensional landscape from the viewer orcamera point. In contrast, the horizontal perspective image appearsdistorted when viewing head on, but this distortion has to be preciselyrendered so that when viewing at a precise location, the horizontalperspective produces a three dimensional illusion.

The present invention multi-plane display system promotes horizontalperspective projection viewing by providing the viewer with the means toadjust the displayed images to maximize the illusion viewing experience.By employing the computation power of the microprocessor and a real timedisplay, the horizontal perspective display of the present invention isshown in FIG. 7, comprising a real time electronic display 100 capableof re-drawing the projected image, together with a viewer's input device102 to adjust the horizontal perspective image. By re-display thehorizontal perspective image so that its projection eyepoint coincideswith the eyepoint of the viewer, the horizontal perspective display ofthe present invention can ensure the minimum distortion in rendering thethree dimension illusion from the horizontal perspective method. Theinput device can be manually operated where the viewer manually inputshis or her eyepoint location, or change the projection image eyepoint toobtain the optimum three dimensional illusion. The input device can alsobe automatically operated where the display automatically tracks theviewer's eyepoint and adjust the projection image accordingly. Themulti-plane display system removes the constraint that the viewerskeeping their heads in relatively fixed positions, a constraint thatcreate much difficulty in the acceptance of precise eyepoint locationsuch as horizontal perspective or hologram display.

The horizontal perspective display system, shown in FIG. 8, can furthera computation device 110 in addition to the real time electronic displaydevice 100 and projection image input device 112 providing input to thecomputational device 110 to calculating the projectional images fordisplay to providing a realistic, minimum distortion three dimensionalillusion to the viewer by coincide the viewer's eyepoint with theprojection image eyepoint. The system can further comprise an imageenlargement/reduction input device 115, or an image rotation inputdevice 117, or an image movement device 119 to allow the viewer toadjust the view of the projection images.

The input device can be operated manually or automatically. The inputdevice can detect the position and orientation of the viewew eyepoint,to compute and to project the image onto the display according to thedetection result. Alternatively, the input device can be made to detectthe position and orientation of the viewer's head along with theorientation of the eyeballs. The input device can comprise an infrareddetection system to detect the position the viewer's head to allow theviewer freedom of head movement. Other embodiments of the input devicecan be the triangulation method of detecting the viewer eyepointlocation, such as a CCD camera providing position data suitable for thehead tracking objectives of the invention. The input device can bemanually operated by the viewer, such as a keyboard, mouse, trackball,joystick, or the like, to indicate the correct display of the horizontalperspective display images.

The head or eye-tracking system can comprise a base unit and ahead-mounted sensor on the head of the viewer. The head-mounted sensorproduces signals showing the position and orientation of the viewer inresponse to the viewer's head movement and eye orientation. Thesesignals can be received by the base unit and are used to compute theproper three dimensional projection images. The head or eye trackingsystem can be infrared cameras to capture images of the viewer's eyes.Using the captured images and other techniques of image processing, theposition and orientation of the viewer's eyes can be determined, andthen provided to the base unit. The head and eye tracking can be done inreal time for small enough time interval to provide continous viewer'shead and eye tracking.

The multi-plane display system comprises a number of new computerhardware and software elements and processes, and together with existingcomponents creates a horizontal perspective viewing simulator. For theviewer to experience these unique viewing simulations the computerhardware viewing surface is preferrably situated horizontally, such thatthe viewer's line of sight is at a 45° angle to the surface. Typically,this means that the viewer is standing or seated vertically, and theviewing surface is horizontal to the ground. Note that although theviewer can experience hands-on simulations at viewing angles other than45° (e.g. 55°, 30° etc.), it is the optimal angle for the brain torecognize the maximum amount of spatial information in an open spaceimage. Therefore, for simplicity's sake, we use “45°” throughout thisdocument to mean “an approximate 45 degree angle”. Further, whilehorizontal viewing surface is preferred since it simulates viewers'experience with the horizontal ground, any viewing surface could offersimilar three dimensional illusion experience. The horizontalperspective illusion can appear to be hanging from a ceiling byprojecting the horizontal perspective images onto a ceiling surface, orappear to be floating from a wall by projecting the horizontalperspective images onto a vertical wall surface.

The viewing simulations are generated within a three dimensionalgraphics view volume, both situated above and below the physical viewingsurface. Mathematically, the computer-generated x, y, z coordinates ofthe Angled Camera point form the vertex of an infinite “pyramid”, whosesides pass through the x, y, z coordinates of the Reference/HorizontalPlane. FIG. 9 illustrates this infinite pyramid, which begins at theAngled Camera point and extending through the Far Clip Plane. Theviewing volume is defined by a Comfort Plane, a plabe on top of theviewing volume, and is appropriately named because its location withinthe pyramid determines the viewer's personal comfort, i.e. how theireyes, head, body, etc. are situated while viewing and interacting withsimulations.

For the viewer to view open space images on their physical viewingdevice it must be positioned properly, which usually means the physicalReference Plane is placed horizontally to the ground. Whatever theviewing device's position relative to the ground, theReference/Horizontal Plane must be at approximately a 45° angle to theviewer's line-of-site for optimum viewing.

One way the viewer might perform this step is to position their CRTcomputer monitor on the floor in a stand, so that theReference/Horizontal Plane is horizontal to the floor. This example usesa CRT-type television or computer monitor, but it could be any type ofviewing device, display screen, monochromic or color display,luminescent, TFT, phosphorescent, computer projectors and other methodof image generation in general, providing a viewing surface atapproximately a 45° angle to the viewer's line-of-sight.

The display needs to know the view's eyepoint to proper display thehorizontal perspective images. One way to do this is for the viewer tosupply the horizontal perspective display with their eye's real-world x,y, z location and line-of-site information relative to the center of thephysical Reference/Horizontal Plane. For example, the viewer tells thehorizontal perspective display that their physical eye will be located12 inches up, and 12 inches back, while looking at the center of theReference/Horizontal Plane. The horizontal perspective display then mapsthe computer-generated Angled Camera point to the viewer's eyepointphysical coordinates and line-of-site. Another way is for the viewer tomanually adjusting an input device such as a mouse, and the horizontalperspective display adjust its image projection eyepoint until theproper eyepoint location is experienced by the viewer. Another way wayis using triangulation with infrared device or camera to automaticallylocate the viewer's eyes locations.

FIG. 10 is an illustration of the horizontal perspective display thatincludes all of the new computer-generated and real physical elements asdescribed in the steps above. It also shows that a real-world elementand its computer-generated equivalent are mapped 1:1 and together sharea common Reference Plane. The full implementation of this horizontalperspective display results in a real-time computer-generated threedimensional graphics appearing in open space on and above a viewingdevice's surface, which is oriented approximately 45° to the viewer'sline-of-sight.

The present invention also allows the viewer to move around the threedimensional display and yet suffer no great distortion since the displaycan track the viewer eyepoint and re-display the images correspondingly,in contrast to the conventional pior art three dimensional image displaywhere it would be projected and computed as seen from a singular viewingpoint, and thus any movement by the viewer away from the intendedviewing point in space would cause gross distortion.

The display system can further comprise a computer capable ofre-calculate the projected image given the movement of the eyepointlocation. The horizontal perspective images can be very complex, tediousto create, or created in ways that are not natural for artists orcameras, and therefore require the use of a computer system for thetasks. To display a three-dimensional image of an object with complexsurfaces or to create an animation sequences would demand a lot ofcomputational power and time, and therefore it is a task well suited tothe computer. Three dimensional capable electronics and computinghardware devices and real-time computer-generated three dimensionalcomputer graphics have advanced significantly recently with markedinnovations in visual, audio and tactile systems, and have producingexcellent hardware and software products to generate realism and morenatural computer-human interfaces.

The multi-plane display system of the present invention are not only indemand for entertainment media such as televisions, movies, and videogames but are also needed from various fields such as education(displaying three-dimensional structures), technological training(displaying three-dimensional equipment). There is an increasing demandfor three-dimensional image displays, which can be viewed from variousangles to enable observation of real objects using object-like images.The horizontal perspective display system is also capable of substitutea computer-generated reality for the viewer observation. The systems mayinclude audio, visual, motion and inputs from the user in order tocreate a complete experience of three dimensional illusion.

The input for the horizontal perspective system can be two dimensionalimage, several images combined to form one single three dimensionalimage, or three dimensional model. The three dimensional image or modelconveys much more information than that a two dimensional image and bychanging viewing angle, the viewer will get the impression of seeing thesame object from different perspectives continuously.

The multi-plane display system can further provide multiple views or“Multi-View” capability. Multi-View provides the viewer with multipleand/or separate left-and right-eye views of the same simulation.Multi-View capability is a significant visual and interactiveimprovement over the single eye view. In Multi-View mode, both the lefteye and right eye images are fused by the viewer's brain into a single,three-dimensional illusion. The problem of the discrepancy betweenaccommodation and convergence of eyes, inherent in stereoscopic images,leading to the viewer's eye fatigue with large discrepancy, can bereduced with the horizontal perspective display, especially for motionimages, since the position of the viewer's gaze point changes when thedisplay scene changes.

In Multi-View mode, the objective is to simulate the actions of the twoeyes to create the perception of depth, namely the left eye and theright right sees slightly different images. Thus Multi-View devices thatcan be used in the present invention include methods with glasses suchas anaglyph method, special polarized glasses or shutter glasses,methods without using glasses such as a parallax stereogram, alenticular method, and mirror method (concave and convex lens).

In anaglyph method, a display image for the right eye and a displayimage for the left eye are respectively superimpose-displayed in twocolors, e.g., red and blue, and observation images for the right andleft eyes are separated using color filters, thus allowing a viewer torecognize a stereoscopic image. The images are displayed usinghorizontal perspective technique with the viewer looking down at anangle. As with one eye horizontal perspective method, the eyepoint ofthe projected images has to be coincide with the eyepoint of the viewer,and therefore the viewer input device is essential in allowing theviewer to observe the three dimensional horizontal perspective illusion.From the early days of the anaglyph method, there are much improvementssuch as the spectrum of the red/blue glasses and display to generatemuch more realizm and comfort to the viewers.

In polarized glasses method, the left eye image and the right eye imageare separated by the use of mutually extinguishing polarizing filterssuch as orthogonally linear polarizer, circular polarizer, ellipticalpolarizer. The images are normally projected onto screens withpolarizing filters and the viewer is then provided with correspondingpolarized glasses. The left and right eye images appear on the screen atthe same time, but only the left eye polarized light is transmittedthrough the left eye lens of the eyeglasses and only the right eyepolarized light is transmitted through the right eye lens.

Another way for stereocopic display is the image sequential system. Insuch a system, the images are displayed sequentially between left eyeand right eye images rather than superimposing them upon one another,and the viewer's lenses are synchronized with the screen display toallow the left eye to see only when the left image is displayed, and theright eye to see only when the right image is displayed. The shutteringof the glasses can be achieved by mechanical shuttering or with liquidcrystal electronic shuttering. In shuttering glass method, displayimages for the right and left eyes are alternately displayed on a CRT ina time sharing manner, and observation images for the right and lefteyes are separated using time sharing shutter glasses which areopened/closed in a time sharing manner in synchronism with the displayimages, thus allowing an observer to recognize a stereoscopic image.

Other way to display stereoscopic images is by optical method. In thismethod, display images for the right and left eyes, which are separatelydisplayed on a viewer using optical means such as prisms, mirror, lens,and the like, are superimpose-displayed as observation images in frontof an observer, thus allowing the observer to recognize a stereoscopicimage. Large convex or concave lenses can also be used where two imageprojectors, projecting left eye and right eye images, are providingfocus to the viewer's left and right eye respectively. A variation ofthe optical method is the lenticular method where the images form oncylindrical lens elements or two dimensional array of lens elements.

FIG. 11 is a horizontal perspective display focusing on how thecomputer-generated person's two eye views are projected onto theHorizontal Plane and then displayed on a stereoscopic 3D capable viewingdevice. FIG. 11 represents one complete display time period. During thisdisplay time period, the horizontal perspective display needs togenerate two different eye views, because in this example thestereoscopic 3D viewing device requires a separate left- and right-eyeview. There are existing stereoscopic 3D viewing devices that requiremore than a separate left- and right-eye view, and because the methoddescribed here can generate multiple views it works for these devices aswell.

The illustration in the upper left of FIG. 11 shows the Angled Camerapoint for the right eye after the first (right) eye-view to begenerated. Once the first (right) eye view is complete, the horizontalperspective display starts the process of rendering thecomputer-generated person's second eye (left-eye) view. The illustrationin the lower left of FIG. 11 shows the Angled Camera point for the lefteye after the completion of this time. But before the rendering processcan begin, the horizontal perspective display makes an adjustment to theAngled Camera point. This is illustrated in FIG. 11 by the left eye's xcoordinate being incremented by two inches. This difference between theright eye's x value and the left eye's x+2″ is what provides thetwo-inch separation between the eyes, which is required for stereoscopic3D viewing. The distances between people's eyes vaty but in the aboveexample we are using the average of 2 inches. It is also possible forthe view to supply the horizontal perspective display with theirpersonal eye separation value. This would make the x value for the leftand right eyes highly accurate for a given viewer and thereby improvethe quality of their stereoscopic 3D view.

Once the horizontal perspective display has incremented the AngledCamera point's x coordinate by two inches, or by the personal eyeseparation value supplied by the viewer, the rendering continues bydisplaying the second (left-eye) view.

Depending on the stereoscopic 3D viewing device used, the horizontalperspective display continues to display the left- and right-eye images,as described above, until it needs to move to the next display timeperiod. An example of when this may occur is if the bear cub moves hispaw or any part of his body. Then a new and second Simulated Image wouldbe required to show the bear cub in its new position. This new SimulatedImage of the bear cub, in a slightly different location, gets renderedduring a new display time period. This process of generating multipleviews via the nonstop incrementing of display time continues as long asthe horizontal perspective display is generating real-time simulationsin stereoscopic 3D.

By rapidly display the horizontal perspective images, three dimensionalillusion of motion can be realized. Typically, 30 to 60 images persecond would be adequate for the eye to perceive motion. For stereocopy,the same display rate is needed for superimposed images, and twice thatamount would be needed for time sequential method.

The display rate is the number of images per second that the displayuses to completely generate and display one image. This is similar to amovie projector where 24 times a second it displays an image. Therefore,1/24 of a second is required for one image to be displayed by theprojector. But the display time could be a variable, meaning thatdepending on the complexity of the view volumes it could take 1/12 or ½a second for the computer to complete just one display image. Since thedisplay was generating a separate left and right eye view of the sameimage, the total display time is twice the display time for one eyeimage.

The present invention further discloses a Multi-Plane display comprisinga horizontal perspective display together with a non-horizontal centralperspective display. FIG. 12 illustrates an example of the presentinvention Multi-Plane display in which the Multi-Plane display is acomputer monitor that is approximately “L” shaped when open. Theend-user views the L-shaped computer monitor from its concave side andat approximately a 45° angle to the bottom of the “L,” as shown in FIG.12. From the end-user's point of view the entire L-shaped computermonitor appears as one single and seamless viewing surface. The bottom Lof the display, positioned horizontally, shows horizontal perspectiveimage, and the other branch of the L display shows central perspectiveimage. The edge is the two display segments is preferably smoothlyjoined and can also having a curvilinear projection to connect the twodisplays of horizontal perspective and central perspective.

The Multi-Plane display can be made with one or more physical viewingsurfaces. For example, the vertical leg of the “L” can be one physicalviewing surface, such as flat panel display, and the horizontal leg ofthe “L” can be a separate flat panel display. The edge of the twodisplay segments can be a non-display segment and therefore the twoviewing surface are not continuous. Each leg of a Multi-Plane display iscalled a viewing plane and as you can see in the upper left of FIG. 25there is a vertical viewing plane and a horizontal viewing plane where acentral perspective image is generated on the vertical plane and ahorizontal perspective image is generated on the horizontal plane, andthen blend the two images where the planes meet, as illustrated in thelower right of FIG. 12.

FIG. 12 also illustrates that a Multi-Plane display is capable ofgenerating multiple views. Meaning that it can display single-viewimages, i.e. a one-eye perspective like the simulation in the upperleft, and/or multi-view images, i.e. separate right and left eye viewslike the simulation in the lower right. And when the L-shaped computermonitor is not being used by the end-user it can be closed and look likethe simulation in the lower left.

FIG. 13 is a simplified illustration of the present inventionMulti-Plane display. In the upper right of FIG. 13 is an example of asingle-view image of a bear cub that is displayed on an L-shapedcomputer monitor. Normally a single-view or one eye image would begenerated with only one camera point, but as you can see there are atleast two camera points for the Multi-Plabe display even though this isa single-view example. This is because each viewing plane of aMulti-Plane device requires its own rendering perspective. One camerapoint is for the horizontal perspective image, which is displayed on thehorizontal surface, and the other camera point is for the centralperspective image, which is displayed on the vertical surface.

To generate both the horizontal perspective and central perspectiveimages requires the creation of two camera eyepoints (which can be thesame or different) as shown in FIG. 13 for two different and separatecamera points labeled OSI and CPI. The vertical viewing plane of theL-shaped monitor, as shown at the bottom of FIG. 13, is the displaysurface for the central perspective images, and thus there is a need todefine another common reference plane for this surface. As discussedabove, the common reference plane is the plane where the images aredisplay, and the computer need to keep track of this plane for thesynchronization of the locations of the displayed images and the realphysical locations. With the L-shaped Multi-Plane device and the twodisplay surfaces, the Simulation can to generate the three dimansionalimages, a horizontal perspective image using (OSI) camera eyepoint, anda central perspective image using (CPI) camera eyepoint.

The multi-plane display system can further include a curvilinearconnection display section to blend the horizontal perspective and thecentral perspective images together at the location of the seam in the“L,” as shown at the bottom of FIG. 13. The multi-plane display systemcan continuously update and display what appears to be a single L-shapedimage on the L-shaped Multi-Plane device.

Furthermore, the multi-plane display system can comprise multipledisplay surfaces together with multiple curvilinear blending sections asshown in FIG. 14. The multiple display surfaces can be a flat wall,multiple adjacent flat walls, a dome, and a curved wraparound panel.

The present invention multi-plane display system thus can simultaneouslyprojecting a plurality of three dimensional images onto multiple displaysurfaces, one of which is a horizontal perspective image. Further, itcan be a stereoscopic multiple display system allowing viewers to usetheir stereoscopic vision for three dimensional image presentation.

Since the multi-plane display system comprises at least two displaysurfaces, various requirements need to be addressed to ensure highfidelity in the three dimensional image projection. The displayrequirements are typically geometric accuracy, to ensure that objectsand features of the image to be correctly positioned, edge matchaccuracy, to ensure continuity between display surfaces, no blendingvariation, to ensure no variation in luminance in the blending sectionof various display surfaces, and field of view, to ensure a continuousimage from the eyepoint of the viewer.

Since the blending section of the multi-plane display system ispreferably a curve surface, some distortion correction could be appliedin order for the image projected onto the blending section surface toappear correct to the viewer. There are various solutions for providingdistortion correction to a display system such as using a test patternimage, designing the image projection system for the specific curvedblending display section, using special video hardware, utilizing apiecewise-linear approximation for the curved blending section. Stillanother distortion correction solution for the curve surface projectionis to automatically computes image distortion correction for any givenposition of the viewer eyepoint and the projector.

Since the multi-plane display system comprises more than one displaysurface, care should be taken to minimize the seams and gaps between theedges of the respective displays. To avoid seams or gaps problem, therecould be at least two image generators generating adjacent overlappedportions of an image. The overlapped image is calculated by an imageprocessor to ensure that the projected pixels in the overlapped areasare adjusted to form the proper displayed images. Other solutions are tocontrol the degree of intensity reduction in the overlapping to create asmooth transition from the image of one display surface to the next.

For realistic three dimensional display, binaural or three dimensionalaudio simulation is also included. The present invention also providethe means to adjust the binaural or 3D audio to ensure proper soundsimulation.

Similar to vision, hearing using one ear is called monoaural and hearingusing two ears is called binaural. Hearing can provide the direction ofthe sound sources but with poorer resolution than vision, the identityand content of a sound source such as speech or music, and the nature ofthe environment via echoes, reverberation such as a normal room or anopen field.

The head and ears, and sometime the shoulder, function as an antennasystem to provide information about the location, distance andenvironment of the sound sources. The brain can interprete properly thevarious kinds of sound arriving at the head such as direct sounds,diffractive sounds around the head and by interaction with the outerears and shoulder, different sound amplitudes and different arrival timeof the sounds. These acoustic modifications are called ‘sound cues’ andserve to provide us the directional acoustis information of the sounds.

Basically, the sound cues are related to timing, volume, frequency andreflection. In timing cues, the ears recognize the time the soundarrives and assume that the sound comes from the closest source.Further, with two ears separated about 8 inches apart, the delay of thesound reaching one ear with respect to the other ear can give a cueabout the location of the sound source. The timing cue is stronger thanthe level cue in the sense that the listener locates the sound based onthe first wave that reaches the ear, regardless of the loudness of anylater arriving waves. In volume (or level) cues, the ears recognize thevolume (or loudness) of the sound and assume that the sound coming fromthe loudest direction. With the binaural (two ears) effect, theamplitude difference between the ears is a strong cue for thelocalization of the sound source. In frequency (or equalization) cues,the ears recognize the frequency balance of the sound as it arrives ineach ear since frontal sounds are directed into the eardrums, while rearsounds bounce off the external ear and thus having a high frequency rolloff. In reflection cue, the sound bounces off various surfaces and areeither dispersed or absorbed in varying degrees before reaching the earsmultiple times. This reflections off the walls of the room and theforeknowledge of the difference between the way various floor coveringssound also contribute to localization. In addition, the body, especiallythe head, can move relative to the sound source to help in locate thesound.

The above various sound cues are scientifically classified into threetypes of spatial hearing cues: interaural time differences (ITDs),interaural level differences (ILDs), and head-related transfer functions(HRTFs). ITDs relate to the time for a sound to reach the ears and thetime difference for reaching both ears. ILDs refer to the amplitude inthe frequency spectrum of sound reaching the ears and also the amplitudedifferences of the sound frequencies as heard in both ears. HRTFs canprovide the perception of distance by the changes in the timbre anddistance dependencies, the time delay and directions of direct sound andreflections in echoic environments.

The HRTFs are a collection of spatial cues for a particular listener,including ITDs, ILDs and the reflections, diffractions and dampingcaused by the listener's body, head, outer ears and shoulder. Theexternal ear, or pinna, has a significant contribution to the HRTFs.Higher frequency sounds are filtered by the pinna to provide the brain away as to perceive the lateral position, or azimuth, and elevation ofthe sound source since the response of the pinna filter is highlydependent on the overall direction of the sound source. The head canaccount for a reduced amplitude of various frequencies of the soundssince the sound has to go through or around the head in order to reachthe ear. The overall effects of head shadowing contribute to theperception of linear distance and direction of a sound source. Further,sound frequencies in the range of 1-3 kHz are reflected from theshoulder to produce echoes representing a time delay dependent on theelevation of the sound source. The reflections from surfaces in theworld and the reverberation also seem to affect the localizationjudgement of sound distance and direction.

In addition to these cues, the movement of the head to help in locatethe location of a sound source is a key factor, together with the visionto confirm the sound direction. For a 3D immersion, all mechanisms tolocalize the sounds are always in play and should normally agree. Ifnot, there would be some discomfort and confusion.

Although we can hear with one ear, hearing with two ears is clearlybetter. Many of the sound cues are related to the binaural perceptiondepending on both the relative loudness of sound and the relative timeof arrival of sound at each ear. And thus the binaural performance isclear superior for the localization of single or multiple sound sourcesand for the formation of the room environment, for the separation ofsignals coming from multiple incoherent and coherent sound sources; andthe enhancement of a chosen signal in a reverberant environment.

Mathematically speaking, HRTF is the frequency response of the soundwaves as received by the ears. By measuring the HRTF of a particularlistener, and by synthesised electronically using digital signalprocessing, the sounds can be delivered to a listener's ears viaheadphones or loudspeakers to create a virtual sound image in threedimensions.

The sound transformation to the ear canal, i.e. HRTF frequency response,can be measured accurately by using small microphones in the ear canals.The measured signal is then processed by a computer to derive the HRTFfrequency responses for the left and right ears corresponding to thesound source location.

Thus a 3D audio system works by using the measured HRTFs as the audiofilters or equalizers. When a sound signal is processed by the HRTFsfilters, the sound localization cues are reproduced, and the listenershould perceive the sound at the location specified by the HRTFs. Thismethod of binaural synthesis works extremely well when the listener'sown HRTFs are used to synthesize the localization cues. However,measuring HRTFs is a complicated procedure, so 3D audio systemstypically use a single set of HRTFs previously measured from aparticular human or manikin subject. Thus the HRTF sometimes needs to bechanged to accurately respond to a perticular listener. The tuning of aHRTF function can be accomplished by providing various sound sourcelocations and environments and asking the listener to identify.

A 3D audio system should provide the ability for the listener to definea three-dimensional space, to position multiple sound sources and thatlistener in that 3D space, and to do it all in real-time, orinteractively. Beside 3D audio system, other technologies such stereoextension and surround sound could offer some aspects of 3D positioningor interactivity.

Extended stereo processes an existing stereo (two channel) soundtrack toadd spaciousness and to make it appear to originate from outside theleft/right speaker locations through fairly straight-forward methods.Some of the characteristics of the extended stereo technology includethe size of the listening area (called sweet spot), the amount ofspreading of stereo images, the amount of tonal changes, the amount oflost stereo panning information, and the ability to achieve effect onheadphones as well as speakers.

The surround sound create a larger-than-stereo sound stage with asurround sound 5-speaker setup. Additionally, virtual surround soundsystems use 3D audio technology to create the illusion of five speakersemanating from a regular set of stereo speakers, therefore enabling asurround sound listening experience without the need for a five speakersetup. The characteristics of the surround sound technology include thepresentation accuracy, the clarity of spatial imaging, and the size ofthe listening area For better 3D audio system, audio technology needs tocreate a life-like listening experience by replicating the 3D audio cuesthat the ears hear in the real world for allowing non-interactive andinteractive listening and positioning of sounds anywhere in thethree-dimensional space surrounding a listener.

The head tracker function is also very important to provide perceptualroom constancy to the listener. In other words, when the listener movetheir heads around, the signals would change so that the perceivedauditory world maintain its spatial position.

To this end, the simulation system needs to know the head position inorder to be able to control the binaural impulse responses adequately.Head position sensors have therefore to be provided. The impression ofbeing immersed is of particular relevance for applications in thecontext of virtual reality.

A replica of a sound field can be produced by putting an infinite numberof microphones everywhere. After being stored on a recorder with aninfinite number of channels, this recording can then be played backthrough an infinite number of point-source loudspeakers, each placedexactly as its corresponding microphone was placed.

As the number of microphones and speakers is reduced, the quality of thesound field being simulated suffers. By the time we are down to twochannels, height cues have certainly been lost and instead of a stagethat is audible from anywhere in the room we find that sources on thestage are now only localizable if we listen along a line equidistantfrom the last two remaining speakers and face them.

However, only two channels should be adequate, since if we deliver theexact sound required to simulate a live performance at the entrance toeach ear canal, then since we only have two ear canals, we should onlyneed to generate two such sound fields. In other words, since we canhear three-dimensionally in the real world using just two ears, it mustbe possible to achieve the same effect from just two speakers or a setof headphones.

Headphone reproduction is thus differed from loudspeaker reproductionsince headphone microphones should be spaced about seven inches apartfor a normal ear separation, and loudspeaker microphones separationshould be about seven feet apart. Further loudspeakers suffer fromcrosstalk and therefore some signal conditioning such as crosstalkcancellation will be needed for 3D loudspeaker setup.

Loudspeaker 3D audio systems are extremely effective in desktopcomputing environments. This is because there is usually only a singlelistener (the computer user) who is almost always centered between thespeakers and facing forward towards the monitor. Thus, the primary usergets the full 3D effect because the crosstalk is properly cancelled. Intypical 3D audio applications, like video gaming, friends may gatheraround to watch. In this case, the best 3D audio effects are heard byothers when they are also centered with respect to the loudspeakers.Off-center listeners may not get the full effect, but they still hear ahigh quality stereo program with some spatial enhancements.

To achieve 3D audio, the speakers are typically arranged surrounding thelistener in about the same horizontal plane, but could be arranged tocompletely surround the listener, from the ceiling to the floor to thesurrounding walls. Optionally, the speakers can also be put on theceiling, on the floor, arranged in an overhead dome configuration, orarranged in a vertical wall configuration. Further, beam transmittedspeakers can be used instead of headphone. Beam transmitted speakeroffers the freedom of movement for the listener and without thecrosstalk between speakers since beam transmitted speaker provide atight beam of sound.

Generally, a minimum of four loudspeakers are required to achieve aconvincing 3-D audio experience, while some researchers are using twentyor more speakers in an anechoic chamber to recreate acousticenvironments with much greater precision.

The main advantages of multi-speaker playback are:

-   -   There is no dependence on the individual subject's HRTF, since        the sound field is created without any reference to individual        listeners.    -   The subject is free to turn their head, and even move about        within a limited range.    -   In some cases, more than one subject can listen to the system        simultaneously.

Many crosstalk cancellers are based on a highly simplified model ofcrosstalk, for example modeling crosstalk as a simple delay andattenuation process, or a delay and a lowpass filter. Other crosstalkcancellers have been based on a spherical head model. As with binauralsynthesis, crosstalk cancellation performance is ultimately limited bythe variation in the size and shape of human heads.

3D audio simulation can be accomplished by the following steps:

-   -   Input the characteristics of the acoustic space.    -   Determine the sequence of sound arrivals that occur at the        listening position. Each sound arrival will have the following        characteristics: (a) time of arrival, based on the distance        travelled by the echo-path, (b) direction of arrival, (c)        attenuation (as a function of frequency) of the sound due to the        absorption properties of the surfaces encountered by the        echo-path.    -   Compute the impulse response of the acoustic space incorporating        the multiple sound arrivals.    -   The results from the FIR filter are played back to a listener.        In the case where the impulse responses were computed using a        dummy head response, the results are played over headphones to        the listener. In this case, the equalisation required for the        particular headphones is also applied.

The simulation of an acoustic environment involves one or more of thefollowing functions:

-   -   Processing an audio source input and presenting it to the        subject through a number of loudspeakers (or headphones) with        the intention of making the sound source appear to be located at        a particular position in space.    -   Processing multiple input audio sources in such a way that each        source is independently located in space around the subject.    -   Enhanced processing to simulate some aspects of the room        acoustics, so that the user can acoustically sense the size of        the room and the nature of the floor and wall coverings.    -   The capability for the subject to move (perhaps within a limited        range) and turn his/her head so as to focus attention on some        aspects of the sound source characteristics or room acoustics.

Binaural simulation is generally carried out using the sound sourcematerial free from any unwanted echoes or noise. The sound sourcematerial can then be replayed to a subject, using the appropriate HRTFfilters, to create the illusion that the source audio is originatingfrom a particular direction. The HRTF filtering is achieved by simplyconvolving the audio signal with the pair of HRTF responses (one HRTFfilter for each channel of the headphone).

The eyes and ears often perceive an event at the same time. Seeing adoor close, and hearing a shutting sound, are interpreted as one eventif they happen synchronously. If we see a door shut without a sound, orwe see a door shut in front of us, and hear a shutting sound to theleft, we get alarmed and confused. In another scenario, we might hear avoice in front of us, and see a hallway with a corner; the combinationof audio and visual cues allows us to figure out that a person might bestanding around the corner. Together, synchronized 3D audio and 3Dvisual cues provide a very strong immersion experience.

Both 3D audio and 3D graphics systems can be greatly enhanced by suchsynchronization.

Improved playback through headphones can be achieved through the use ofhead tracking. This technique makes use of continuous measurements ofthe orientation of a subject's head, and adapts the audio signals beingfed to the headphones appropriately.

Binaural signal should allow a subject to easily discriminate betweenleft and right sound source locations easily, but the ability todiscriminate between front and back, and high and low sound sources isgenerally only possible if head movement is permitted. Whilst multiplespeaker playback methods solve this problem to a large degree, there arestill many applications where headphone playback is preferable, and headtracking can then be used as a valuable tool for improving the qualityof the 3-D playback.

The simplest form of head tracking binaural system is one which simplysimulates anechoic HRTFs, and changes the HRTF functions rapidly inresponse to the subjects head movements. This HRTF switching can beachieved through a lookup table, with interpolation used to resolveangles that are not represented in the HRTF table.

Simulation of room acoustics over headphones with head tracking becomesmore difficult because the direction of arrival of the early reflectionsis also important in making the result sound realistic. Many researchersbelieve that the echoes in the reverberant tail of the room response aregenerally so diffuse that there is no requirement for this part of theroom response to be tracked with the subject's head movements.

An important feature of any head tracking playback system is the delayfrom the subject head movement to the change in the audio response atthe headphones. If this delay is excessive, the subject can experience aform of virtual motion sickness and general disorientation.

Audio cues change dramatically when a listener tilts or rotates his orher head. For example, quickly turning the head 90 degrees to look tothe side is the equivalent of a sound traveling from the listener's sideto the front in a split second. We often use head motion to track soundsor to search for them. The ears alert the brain about an event outsideof the area that the eyes are currently focused on, and we automaticallyturn to redirect our attention. Additionally, we use head motion toresolve ambiguities: a faint, low sound could be either in front or backof us, so we quickly and sub-consciously turn our head a small fractionto the left, and we know if the sound is now off to the right, it is inthe front, otherwise it is in the back. One of the reasons whyinteractive audio is more realistic than pre-recorded audio(soundtracks) is the fact that the listeners head motion can be properlysimulated in an interactive system (using inputs from a joystick, mouse,or head-tracking system).

The HRTF function are performed using digital signal processing (DSP)hardware for real time performance. Typical feature of DSP are that thedirect sound must be processed to give the correct amplitude andperceived direction, the early echoes must arrive at the listener withappropriate time, amplitude and frequency response to give theperception of the size of the spaces (as well as the acoustic nature ofthe room surfaces), and the late reverberation must be natural andcorrectly distributed in 3-D around the listener. The relative amplitudeof the direct sound compared to the remainder of the room response helpsto provide the sensation of distance.

Thus 3D audio simulation can provides a binaural gain so that the exactsame audio content is more audible and intelligible in the binauralcase, because the brain can localize and therefore “single out” thebinaural signal, while the non-binaural signal gets washed into thenoise. Further the listener would still be able to tune into andunderstand individual conversations, because they are still spatiallyseparated, and “amplified by” binaural gain, an effect called thecocktail party effct. Binaural simulation also can provide fasterreaction time because such a signal mirrors the ones received in thereal world. In addition, binaural signals can convey positionalinformation: a binaural radar warning sound can warn a user about aspecific object that is approaching (with a sound that is unique to thatobject), and naturally indicate where that object is coming from. Alsolistening to binaural simulation can be less fatigue since we are usedto hearing sounds that originate outside of their heads, as is the casewith binaural signals. Mono or stereo signals appear to come from insidea listener's head when using headphones, and produce more strain than anatural sounding, binaural signal. An lastly, 3D binaural simulation canprovide an increased perception and immersion in higher quality 3Denvironment when visuals are shown in synch with binaural sound.

1. A method of three dimensional image display by horizontal perspectiveprojection and 3-D audio system projection, the horizontal perspectiveprojection comprising a display of horizontal perspective imagesaccording to a predetermined projection eyepoint, and the 3-D audiosystem projection comprising providing 3-D sound according to apredertermined projection earpoint, the method comprising the steps of:detecting an eyepoint location of a viewer; displaying a horizontalperspective image using the detected eyepoint location as the projectioneyepoint; and projecting a 3-D sound to the viewer using the eyepointlocation as the projection earpoint.
 2. A method as in claim 2 whereinthe 3-D sound comprises two sound channels and a HRTF (head relatedtransfer function) filter.
 3. A method as in claim 2 wherein the 3-Dsound comprises a 3-D loudspeaker audio system or a 3-D headphone audiosystem.
 4. A method of three dimensional image display by horizontalperspective projection and 3-D audio system projection, the horizontalperspective projection comprising a display of horizontal perspectiveimages according to a predetermined projection eyepoint, and the 3-Daudio system projection comprising providing 3-D sound according to apredertermined projection earpoint, the method comprising the steps of:continuously scanning to detect an eyepoint location of a viewer;calculating a new horizontal perspective image using the detectedeyepoint location as the projection eyepoint; displaying the new image;calculating a new 3-D sound using the detected eyepoint location as theprojection earpoint; and displaying the new 3-D sound.
 5. A method as inclaim 4 wherein the 3-D sound comprises two sound channels and a HRTF(head related transfer function) filter.
 6. A method as in claim 4wherein the 3-D sound comprises a 3-D loudspeaker audio system or a 3-Dheadphone audio system.
 7. A method as in claim 4 wherein the horizontalperspective image is stereoscopic images.
 8. A method as in claim 4wherein the horizontal perspective image is calculated from a flat twodimensional picture.
 9. A method as in claim 4 wherein the horizontalperspective image is calculated from a three dimensional model.
 10. Amethod as in claim 4 wherein detecting a viewer eyepoint location is bymanually inputting the location through a manual input device.
 11. Amethod as in claim 10 wherein the manual input device is a computerperipheral or a wireless computer peripheral.
 12. A method as in claim10 wherein the manual input device is selected from a group consisted ofa keyboard, a stylus, a keypad, a computer mouse, a computer trackball,a tablet, a pointing device.
 13. A method as in claim 4 wherein thedetection of a viewer eyepoint location is through an automatic inputdevice whereby the automatic input device automatically extracts theeyepoint location from the viewer.
 14. A method as in claim 13 whereinthe automatic input device is selected from a group consisted ofradio-frequency tracking device, infrared tracking device, cameratracking device.
 15. A method as in claim 4 further comprising the stepof manipulating the image.
 16. A method as in claim 4 wherein themanipulation of the image comprises the modification of the displayedimage or the generation of a new image.
 17. A method as in claim 4wherein the modification of the displayed image comprises the movement,zooming or rotation of the image.
 18. A method of image display byhorizontal perspective projection and 3-D audio system projection, thehorizontal perspective projection comprising a display of horizontalperspective images according to a predetermined projection eyepoint, andthe 3-D audio system projection comprising providing 3-D sound accordingto a predertermined projection earpoint, the method comprising the stepsof: displaying a first image onto a first display; continuously scanningto detect an eyepoint location of a viewer; calculating a secondhorizontal perspective image using the detected eyepoint location as theprojection eyepoint; displaying the second image onto a secondhorizontal perspective display; and projecting a 3-D sound to the viewerusing the eyepoint location as the projection earpoint.
 19. A method asin claim 18 further comprising a step of display a third image onto athird curvilinear display, the curvilinear display blending the firstdisplay and the second display.
 20. A method as in claim 18 furthercomprising a step of calculating the third image using the eyepointlocation as the projection eyepoint before displaying.